Code in pre-training data improves LLM performance at non-coding tasks

A study by Cohere shows that pre-training LLMs on code improves their abilities in non-coding tasks, including world knowledge and reasoning.